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(54) Virtual Private Network for encrypted firewall 

(57) A system (10) for regulating the flow of messages through a firewall (18) having a network protocol 
stack, wherein the network protocol stack includes an Internet Protocol (IP) layer where if the message is not 
encrypted, it passes the unencrypted message up the network protocol stack to an application level proxy (50), 
and if the message is encrypted, it decrypts the message and passes the decrypted message up the network 
protocol stack to the application level proxy. The step of decrypting the message includes the step of executing 
a process at the IP layer to decrypt the message* 
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At least one drawing originally filed was informal and the print reproduced here is taken from a later filed formal copy. 
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VIRTUAL PRIVATE NETWORK ON APPLICATION GATEWAY 

5 Background of the Invention 

Field of the Invention 

The present invention pertains generally to network communications, and 
in particular to a system and method for securely transferring information 
between firewalls over an unprotected network. 

10 Background Information 

Firewalls have become an increasingly important part of network design. 
Firewalls provide protection of valuable resources on a private network while 
allowing communication and access with systems located on an unprotected 
network such as the Internet In addition, they operate to block attacks on a 

15 private network arriving from the unprotected network by providing a single 
connection with limited services. A well designed firewall limits the security 
problems of an Internet connection to a single firewall computer system. This 
allows an organization to focus their network security efforts on the definition of 
the security policy enforced by the firewall. An example of a firewall is given in 

20 "SYSTEM AND METHOD FOR PROVIDING SECURE INTERNETWORK 
SERVICES" by Boebert et al. (PCT Published Application No. WO 96/13 1 13, 
published on May 2, 1996), the description of which is hereby incorporated by 
reference. Another description of a firewall is provided by Dan Thomsen in 
"Type Enforcement: the new security model", Proceedings: Multimedia: Full- 

25 Service Impact on Business, Education, and the Home, SPIE Vol. 2617, p. 143, 
August 1996. Yet another such system is described in "SYSTEM AND 
METHOD FOR ACHIEVING NETWORK SEPARATION" by Gooderum et al. 
(PCT Published Application No. WO 97/29413, published on August 1 4, 1997), 
the description of which is hereby incorporated by reference. All the above 

30 systems are examples of application level gateways. Application level gateways 
use proxies or other such mechanisms operating at the application layer to 
process traffic through the firewall. As such, they can review not only the 
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message traffic but also message content In addition, they provide 
authentication and identification services, access control and auditing. 

Data to be transferred on unprotected networks like the Internet is 
susceptible to electronic eavesdropping and accidental (or deliberate) corruption. 
S Although a firewall can protect data within a private network from attacks 
launched from the unprotected network, even that data is vulnerable to both 
eavesdropping and corruption when transferred from the private network to an 
external machine. To address this danger, the Internet Engineering Task Force 
(IETF) developed a standard for protecting data transferred between firewalls 

10 over an unprotected network. The Internet Protocol Security (IPSEC) standard 
calls for encrypting data before it leaves the first firewall, and then decrypting 
the data when it is received by the second firewall. The decrypted data is then 
delivered to its destination, usually a user workstation connected to the second 
firewall. For this reason IPSEC encryption is sometimes called fir<rwall-to~ 

1 5 firewall encryption (FFE) and the connection between a workstation connected 
to the first firewall and a client or server connected to the second firewall is 
termed a virtual private network, or VPN. 

The two main components of IPSEC security are data encryption and 
sender authentication. Data encryption increases the cost and time required for 

20 the eavesdropping party to read the transmitted data. Sender authentication 
ensures that the destination system can verify whether or not the encrypted data 
was actually sent from the workstation that it was supposed to be sent from. The 
IPSEC standard defines an encapsulated payload (ESP) as die mechanism used 
to transfer encrypted data. The standard defines an authentication header (AH) 

25 as the mechanism for establishing the sending workstation's identity. 

Through the proper use of encryption, the problems of eavesdropping and 
corruption can be avoided; in effect, a protected connection is established from 
the internal network connected to one firewall through to an internal network 
connected to the second firewall. In addition, IPSEC can be used to provide a 

30 protected connection to an external computing system such as a portable 
personal computer. 



IPSEC encryption and decryption work within the IP layer of the network 
protocol stack. This means that all communication between two IP addresses 
will be protected because all interfirewall communication must go through the IP 
layer. Such an approach is preferable over encryption and decryption at higher 
5 levels in Ac network protocol stack since when encryption is performed at layers 
higher than the IP layer more work is required to ensure that all supported 
communication is properly protected. In addition, since IPSEC encryption is 
handled below the Transport layer, IPSEC can encrypt data sent by any 
application. IPSEC therefore becomes a transparent add-on to such protocols as 
10 TCPandUDP. 

Since, however, IPSEC decryption occurs at the IP layer, it can be 
difficult to port IPSEC to an application level gateway while still maintaining 
control at the proxy over authentication, message content, access control and 
auditing. Although the IPSEC specification in RFC 1825 suggests the use of a 
1 5 mandatory access control mechanism in a multi-level secure (MLS) network to 
compare a security level associated with the message with the security level of 
the receiving process, such an approach provides only limited utility in an 
application level gateway environment In feet, implementations on application 
level gateways to date have simply relied on the feet that themes sage was 
IPSEC-encrypted as assurance that the message is legitimate and have simply 
decoded and forwarded the message to its destine This creates, however, a 
potential chink in the firewall by assuming that the encrypted communication 
has access to all services. 

What is needed is a method of handling IPSEC messages within an 
application level gateway which overcomes the above deficiencies. The method 
should allow control over access by an IPSEC connection to individual services 
within the internal network. 

Summary flf fte Ttiwwri w 
Tie present invention is a system and method for regulating the flow of 
messages through a firewall having a network protocol slack, wherein the 
network protocol stack includes an Internet Protocol (IP) layer, the method 
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comprising the steps of determining, at the IP layer, if a message is encrypted, if 
die message is not encrypted, passing the unencrypted message up the network 
protocol stack to an application level proxy, and if the message is encrypted, 
decrypting the message and passing the decrypted message up the network 
protocol stack to the application level proxy, wherein the step of decrypting the 
message includes the step of executing a procedure at the IP layer to decrypt the 
message. 

According to another aspect of the present invention, a system and 
method is described for authenticating the sender of a message within a 
computer system having a network protocol stack, wherein the network protocol 
stack includes an Internet Protocol (IP) layer, the method comprising the steps of 
detennining, at the IP layer, if the message is encrypted, if the message is 
encrypted, decrypting the message, wherein the step of decrypting the message 
includes the step of executing a procedure at the P layer to decrypt the message, 
15 passing the decrypted message up the network protocol stack to an application 
level proxy, detaining an authentication protocol appropriate for the message, 
and executing the authentication protocol to authenticate the sender of the 
message. 

Brief Description of the Tliiwfaffl 
20 In the following detailed description of example embodiments of the 

invention, reference is made to the accompanying drawings which form a part 
hereof, and which is shown by way of illustration only, specific embodiments in 
which the invention may be practiced. It is to be understood that other 
embodiments may be utilized and structural changes may be made without 
25 departing from the scope of the present invention. 

In the drawings, where like numerals refer to like components throughout 
the several views: 

Figure 1 is a functional block diagram of an application level gateway- 
implemented firewaU-to-fircwall encryption scheme according to the present 
30 invention; 



Figure 2 is a block diagram showing access control checking of both 
encrypted and unencrypted messages in network protocol stack according to the 
present invention; 

Figure 3 is ablock diagram of a representative application level gateway- 
implemented firewall-to-firewall encryption scheme; 

Figure 4 is a block diagram of one embodiment of a network-separated 
protocol stack implementing IPSEC according to the present invention; and 

Figure 5 is a functional block diagram of a firewall-to-workstation 
encryption scheme according to the present invention. 

Description of the Pr eferred Rmhgdirq^ 
In the following detailed description of die preferred embodiment, 
references made to the accompanying drawings which fonn a part hereof, and in 
which is shown by way of illustration specific preferred embodiments in which 
the invention may be practiced. These embodiments are described in sufficient 
detail to enable those skilled in the art to practice the invention, and it is to be 
understood that other embodiments may be utilized and that structural, logical, 
physical, architectural, and electrical changes may be made without departing 
from the spirit and scope of the present invention- the following detailed 
description is, therefore, not to be taken in a limiting sense, and the scope of the 
present invention is defined only by die appended claims and their equivalents. 

A system 1 0 which can be used for fircwaU-to-firewall encryption (FFE) 
is shown in Figure L In Figure I, system 10 includes a workstation 12 
communicating through a firewall 14 to an unprotected network 16 such as the 
Interact System 10 also includes a workstation 20 communicating through a 
firewall 18 to unprotected network 16. In one embodiment, firewall 18 is an 
application level gateway. 

As noted above, IPSEC encryption and decryption work within the IP 
layer of the network protocol stack. This means that all communications 
between two IP addresses will be protected because all interfirewaU 
communication must pass through the IP layer. IPSEC takes the standard 



Internet packet and converts it into a carrier packet. The carrier packet is 
designed to do two things: to conceal the contents of the original packet 
(encryption) and to provide a mechanism by which the receiving firewall can 
verify the source of the packet (authentication), In one embodiment of the 
present invention, each IPSEC carrier packet includes both an authentication 
header used to authenticate die sending machine and an encapsulated payload 
containing encrypted data. The authentication header and the encapsulated 
payload features of IPSEC can, however, be used independently. As required in 
RFC 1 825, DES-CBC is provided for use in encrypting the encapsulated payload 
while the authentication header uses keyed MD5. 

To use IPSEC, you must create a security association (S A) for each 
destination IP address. In one embodiment, each S A contains the following 
information: 

Security Parameters Index (SPI) - The index used to find a SA on 
receipt of an IPSEC datagram. 

Destination IP address - The address used to find the SA and 

trigger use of IPSEC processing on output, 

The peer SPI - The SPI value to put on a IPSEC datagram on 

output 

The peer IP address - The destination IP address to be put into the 

packet header if IPSEC Tunnel mode is used 

The Encryption Security Payload (ESP) algorithm to be used. 

The ESP key to used for decryption of input datagrams. 

The ESP key to used for encryption of output datagrams. 

The authentication (AH) algorithm to be used. 

The AH key to be used for validation of input packets , 

The AH key to be used for generation of the authentication data 

for output datagrams. 



The combination of a given Security Parameter Index and Destination IP 
address uniquely identifies a particular "Security Association.*' In one 



embodiment, the sending firewall uses the sending userid and Destination 
Address to select an appropriate Security Association (and hence SPI value). 
The receiving firewall uses the combination of SPI value and Source address to 
obtain the appropriate Security Association. 

A security association is normally one-way. An authenticated 
communications session between two firewalls will normally have two Security 
Parameter Indexes in use (one in each direction). The combination of a 
particular Security Parameter Index and a particular Destination Address 
uniquely identifies the Security Association. 

More information on the specifics of an IPSEC FFE implementation can 
be obtained from the standards developed by the IPSEC work group and 
documented in Security Architecture for IP (RFC 1825) and in RFC's 1 826- 
1829. 

When a datagram is received from unprotected network 16 or is to be 
transmitted to a destination across unprotected network 16, the firewall must be 
able to determine the algorithms, keys, etc. that must be used to process the 
datagram correctly. In one embodiment, this information is obtained via a 
security association lookup. In one such embodiment, the lookup routine is 
passed several arguments: the source IP address if the datagram is being received 
from network 1 6 or the destination IP address if the datagram is to be transmitted 
across network 16, the SPI, and a flag that is used to indicate whedier the lookup 
is being done to receive or transmit a datagram 

When an IPSEC datagram is received by firewall 18 from unprotected 
network 16, the SPI and source IP address are determined by looking in the 
datagram. In one embodiment a Security Association Database (SADB) stored 
within firewall 18 is searched for the entry with a matching SPI. In one such 
embodiment, security associations can be set up based on network address as 
well as a more granular host address. This allows the network administrator to 
create a security association between two firewalls with only a couple of lines in 
a configuration file on each machine. For such embodiments, the entry in the 
Security Association Database that has both die matching SPI and the longest 
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address match is selected as the SA entry. In another such embodiment, each S A 
has a prefix length value associated with the address. An address match on a SA 
entry means that the addresses match for the number of bits specified by the 
prefix length value. 

5 There are two exceptions to this search process. First, when an SA entry 

is set marked as being dynamic it implies that die user of this S A may not have a 
fixed IP address. In this case the match is fully determined by the SPI value. 
Thus it is necessary that the SPI values for such SA entries be unique in the 
SADB. The second exception is for SA entries marked as tunnel mode entries. 

10 ' In this case it is normally the case that the sending entity will hide its source 
address so that all that is visible on the public wire is the destination address. In 
this case, like in the case where the SA entries are for dynamic IP addresses, the 
search is done exclusively on the basis of the SPL 

When transmitting a datagram across unprotected network 16 the SADB 

IS is searched using only the destination address as an input In this case the entry 
which has the longest address match is selected and returned to the calling 
routine. 

In one embodiment, if firewall 18 receives datagrams which are 
identified as either an ff _PROTO JPSECESP or IP J>ROTOJPSEC_AH 

20 protocol datagram, there must be a corresponding SA in the SADB or else 

firewall 18 will drop the packet and an audit message will be generated. Such an 
occurrence might indicate a possible attack or it might simply be a symptom of 
an erroneous key entry in the Security Association Database. 

In a system such as system 10, application level gateway firewall 18 acts 

25 as a buffer between unprotected network 16 and workstations such as 

workstation 20. Messages coming from unprotected network 1 6 are reviewed 
and a determination is made as to whether execution of an authentication and 
identification protocol is warranted. In contrast to previous systems, system 1 0 
also performs this same determination on IPSEC-encrypted messages. If 

30 desired, the same authentication and identification can be made on messages to 
be transferred from workstation 20 to unprotected network 16. Figure 2 
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illustrates one way of authenticating both encrypted and unencrypted messages 
in a system such as system 10. 

In the system of Figure 2 a network protocol stack 40 includes a physical 
layer 42, an Internet protocol (EP) layer 44, a Transport layer 46 and an 

5 application layer 48. Such a protocol stack exists, for instance on application 
level gateway firewall 18 of Figure 1 . An application executing in application 
layer 48 can communicate to an application executing on another system by 
preparing a message and transmitting it through one of the existing transport 
services executing on transport layer 46. Transport layer 46 in turn uses a 

10 process executing in IP layer 44 to continue the transfer. Physical layer 42 
provides the software needed to transfer data through the communication 
hardware (e.g., a network interface card or a modem). As noted above, IPSEC 
executes within IP layer 44. Encryption and authentication is transparent to the 
host as long as the network administrator has the Security Association Database 

1 5 correctly configured and a key management mechanism is in place on the 
firewall. 

In application level gateway firewall 18, a proxy SO operating within 
application layer 48 processes messages transferred between internal and 
external networks. All network-to-network traffic must pass through one of the 

20 proxies within application layer 48 before being the transfer across networks is 
allowed. A message arriving from external network 16 is examined at IP layer 
44 and an SADB is queried to determine if the source address and SPI are 
associated with an S A. In the embodiment shown in Figure 2, an SADB Master 
copy 52 is maintained in persistent memory at application layer 48 while a copy 

25 54 of SADB is maintained in volatile memory within the kernel. If the message 
is supposed to be encrypted, the message is decrypted based on the algorithm 
and key associated with the particular SA and the message is transferred up 
through transport layer 46 to proxy 50. Proxy 50 examines the source and 
destination addresses and the type of service desired and decides whether 

30 authentication of the sender is warranted If so, proxy 50 initiates an 

authentication protocol. The protocol may be as simple as requesting a user 
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name and password or it may include a challenge/response authentication 
process. Proxy SO also looks to see whether the message coining in was 
encrypted or not and may factor that into whether a particular type of 
authentication is needed. In Telnet, for instance, user name/password 
5 authentication may be sufficient for an FFE link while the security policy may 
dictate that a more stringent challenge/response protocol is needed for 
unencrypted links. In that case, proxy 50 will be a Telnet proxy and it will base 
its authentication protocol on whether the link was encrypted or not. 

Since IPSEC executes within IP layer 44 there is no need for host 

10 firewalls to update their applications. Users that already have IPSEC available 
on their own host machine will, however, have to request that die firewall 
administrator set up SA's in the SADB for their traffic. 

In the embodiment shown in Figure 2, a working copy 54 of the Security 
Association Database consisting of all currently active SA's is kept resident in 

15 memory for ready access by IP layer processing as datagrams are received and 
transmitted. In addition, a working master copy 52 of the SADB is maintained 
in a file in nonvolatile memory. During system startup and initialization 
processing the content of all of the required S A' s in master SADB 52 is added to 
the working copy 54 stored in kernel memory. 

20 In one embodiment, firewall 18 maintains different levels of security on 

internal and external network interfaces. It is desirable for a firewall to have 
different levels of security on both the internal and external interfaces. In one 
embodiment, firewall 18 supports three different levels, numbered 0 through 2. 
These levels provide a simple policy mechanism that controls permission for 

25 both in-bound and out-bound packets. 

Level 0 - do not allow any in-bound or out-bound traffic unless there is a 
security association between the source and destination. 



Level 1 - Allow both in-bound and out-bound non-IPSEC traffic but 
force the use of BPSEC if a SA exists for the address. (To support this firewall 
18 must look for a SA for each in-bound datagram-) 

Level 2 - allow NULL security associations to exist. NULL associations 
S are just like normal security associations, except no encryption or authentication 
transform is performed on in-bound or out-bound packets that correspond to this 
NULL association. With Level 2 enabled, the machine will still receive 
unprotected traffic, but it will not transmit unless Level 1 is enabled. 

The default protection level established when die Security Association 
10 Database (SADB) is initialized at boot time is 1 for in-bound traffic and 2 for 
out-bound traffic. 

An Access Control List, or ACL, is a list of rules that regulate the flow of 
Internet connections through a firewall. These rules control how a firewall's 
servers and proxies will react to connection attempts. When a server or proxy 

15 receives an incoming connection, it performs an ACL check on that coiuiection- 
An ACL check compares a set of parameters associated with the 
connection against a list of ACL rules. The rules determine whether the 
connection is allowed or denied A rule can also have one or more side effects. 
A side effect causes the proxy to change its behavior in some fashion. For 

20 example, a common side effect is to redirect the destination IP address to an 
alternate machine. In addition to IP connection attempts, ACL checks can also 
made on the console logins and on logins made from serial ports. Finally, ACL 
checks can also be made on behalf of IP access devices, such as a Cisco box, 
through the use of the industry standard TACACS+ protocol. 

25 In one embodiment, the ACL is managed by an acid daemon running in 

the kernel of firewalls 10 and 30. The acid daemon receives two types of 
requests, one to query the ACL and one to administer it In one such 
embodiment, the ACL is stored in a relational database such as the Oracle 
database for fast access. By using such a database, query execution is 

30 asynchronous and many queries can be executing concurrently. In addition, 
these types of databases are designed to manipulate long lists of rules quickly 
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and efficiently. These qualities ensure that a given query cannot hang up the 
process that issued the query for any appreciable time (> 1-2 seconds). 

In one such embodiment, the database can hold up to 100,000 users and 
up to 10,000 hosts but can be scaled up to the capacity of the underlying 
5 database engine. The results of an ACL check is cached, allowing repeated 
checks to be turned around very quickly. 

Applications on firewalls 10 and 30 can query acid to determine if a 
given connection attempt should be allowed to succeed In one embodiment, the 
types of applications (i.e. "agents") that can make ACL queries can be divided 
10 into four classes; 

1) Proxies. These allow connections to pass through firewall 10 or 30 in 

order to provide access to a remote service. They include tnauthp 

(authenticated telnet proxy), pftp (FTP proxy), httpp (HTTP proxy), and 

tcpgsp (TCP generic service proxy). 
15 2) Servers. These provide a service on the firewall itself They include ftpd 

andhttpd. 

3) Login agents. Login agent is a program on the firewall that can create a 
Unix shell. It is not considered a server because it cannot receive IP 
connections. One example is Aisr/bin/login when used to create a dialup 

20 session or a console session on firewall 1 0 or 30. Another example is the 

command srole. 

4) Network Access Servers (NAS). NAS is a remote IP access device, 
typically a dialup box manufactured by such companies as Cisco or 
Bridge. The NAS usually provides dialup telnet service and may also 

25 provide SUP or PPP service. 

Proxies, servers, login agents, andNASes make queries to acid to 
determine if a given connection attempt should be allowed to succeed. All of the 
agents except NAS make their queries directly. NAS, because it is remote, must 
communicate via an auxiliary daemon that typically uses an industry standard 

30 protocol such as RADIUS or TACACS+. The auxiliary daemon (e.g., tacradd) 
in turn forwards the query to local acid. 
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As a side effect of the query, acid tells the agent if authentication is 
needed. If no authentication is needed, the connection proceeds immediately. 
Otherwise acid provides (as another side effect) a list of allowed authentication 
methods that the user can choose from. The agent can present a menu of choices 
5 or simply pick the first authentication method by default. Typical authentication 
methods include plain password, SNK DSS, SDI SecurlD, LOCKout DES, and 
LOCKout FORTEZZA. In one embodiment, the list of allowed authentication 
methods varies depending on the host name, user name, time of day, or any 
combination thereof 

10 In the case of a Level 0 policy, it would be safe to assume that all 

incoming traffic is encrypted or authenticated. In the case of Levels 1 through 2 ; 
a determination must be made whether or not a security association exists for a 
given peer. Otherwise an application may believe that in-bound traffic has been 
authenticated when it really has not (That is why it is necessary to look for an 

IS S A on input of each non-IPSEC datagram.) 

In one embodiment, a flag which accompanies the message as it is sent 
from IP layer 44 to proxy SO indicates whether the incoming message was or was 
not encrypted. In another embodiment, proxy 50 accesses Security Association 
Database 54 (the table in the kernel can be queried via an SADB routing socket 

20 (PF-SADB)) to determine whether or not a security association exists for a given 
peer- The SADB socket is much like a routing socket found in the stock BSD 
4.4 kernel (protocol family PF-ROUTE) except that PF-SADB sockets arc used 
to maintain the Security Association Database (SADB) instead of the routing 
table. Because the private keys used for encryption, decryption, and keyed 

25 authentication are stored in this table, access must be strictly prohibited and 
allowed to only administrators and key management daemons. Care must be 
taken when allowing user-level daemons access to /dev/mem or /dev/kmem 
as well, since the keys are stored in kernel memory and could be exposed with 
some creative hacking, 

30 In one embodiment, a command-line tool called sadb is used to support 

the generation and maintenance of in-kemel version 54 of SADB. The primary 
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interface between this tool and the SADB is the PF-SADB socket The kernel 
provides socket processing to receive client requests to add, update, or change 
entries in in-kernel SADB 54. As noted above, the default protection level 
established when the Security Association Database (SADB) is initialized at boot 
S time is 1 for in-bound traffic and 2 for out-bound traffic. This may be changed 
by the use of the sadb command. 

The existing sadb command was derived from the NIST implementation 
of IPSEC As noted above, this tool is much like route in that it uses a special 
socket to pass data structures in and out of die kernel. There are three commands 
10 recognized by the sadb command: get, set, delete. The following simple shell 
script supports adding and removing a single SA entry to SADB 54. It shows 
one embodiment of a parameter order for adding a SA to the SADB. 

# ! /bin/oh 
15 if [ $# -ne 1 ] 
then 

echo "usage: 50 <on>|<off> n >&2 
exit 1 

fi 

20 ONOFF«$l 

addsa ( ) 
{ 

IPADDRESS=$2 
25 PEERADDRESS - 0 .0.0.0 

PREFIXLEN=0 # Num of bits, 0 «> full 32 

bit: match 

LOCALADDRESS-0 .0.0.0 
REALADDR£SS»0 .0.0.0 
30 PORT=0 

PROTOCOLS 
UID-0 

DESALG=1 # I DES-CBC 

IVLEN-4 # bytes 

35 DESKEY=ObObQbObOb0b0b0b 

DESKEYLEN=8 # bytes 

AHALG=1 # 1 = MD5 

AHKEY=30313233343536373031323334353637 

AHKE YLEN= 1 6 # bytes 

40 LOCAL_SPI=$l 
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PEER_SPI=S1 
TUNNEL_MODE= 0 
AHRESULTLEN=4 

COMBINED_MODE~l # On output, 1 = ESP, then 

5 AH; 0 «- AH, then ESP 
DYNAMT CFLAG* 0 

if [ n $ONOFF" = "on" 
then 

10 ./eadb add dst $IPADDRESS SPREFIXLEN $LOCAL_SPI 

$UID $ PEERADDRES S $PEER_SPI ?TONNEL_MODE $ LOCALADDRE SS 
SREALADDRESS $PROTOCOL SPORT $DESALG $IVLEN $DESKEYLEN 
SDESKEY $DESKEYLEN $DESKEY $AHALG $AHKEYLEN $AHKEY 
$AHKEYLEN SAHKEY $AHRESULTLEN $COMBINED_MODE 

15 $ DYNAMI C_FLAG 
else 

./sadb delete dst $IPADDRESS $LOCAL-SPI 

fi 
} 

20 

# Get down to work: 

addsa 500 172.17.128.115 # numberG . sctc , com 

The current status of in-kerne] SADB 54 can be obtained with flic sadb 
25 command- The get option allows dumping the entire SADB or a single entry. In 
one embodiment, the complete dump approach uses /dev/kmem to find the 
information. The information may be presented as follows: 



# sadb get dst 

30 

Local-SPI Address-Family Destination-Addr 
Pref IxJLength UID 

Peer-Address Peer-SPI Transport -Type 
Local -Address Real-Address 
35 Protocol Port 

ESP_Alg_ID ESP_IVEC_Length 

ESP_Enc_Key_length ESP_Enc_ESP_Key 
ESP_Dec_KeyJLength ESP_Dec_ESP_Key 
AH_Alg_ID AH_Data_Length 
40 AH_Gen_Key_Length AH_Gen_Key 

AH_Check_KeyjLength AHjZheckJKey 
Combined_Mode Dynamic_Flag 
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500 INET: number6.sctc.com 0 0 v 

0.0.0.0 500 Transport (0) 0 
5 0.0.0.0 0.0.0.0 

None None 

DES/CBC-RFC1829 (1) 4 

6 ObObObObObObObOb 

8 ObObObObObObObOb 
10 MD5-RFC1828>(1) 4 

16 30313233343536373031323334353637 

16 30313233343536373031323334353637 
ESP+AH(1) 0 

501 INET: spokes . ectc . com 0 0 

15 0.0.0-0 501 Transport (0) 0 

0.0.0.0,0.0.0.0 
None None 

DES/CBORFC1829(l) 4 

8 ObObObObObObObOb 
20 8 ObObObObObObObOb 

MD5-RFC1828(1)4 

16 30313233343536373031323334353637 

16 3031323334353637303 13233343S3637 
ESP+AHU) 0 

25 

End of list. 

When a new entry is added to in-kcxnel SADB 54, the add process first 
checks to see that no existing entry will match the values provided in the new 
30 entry. If no match is found then the entry is added to the end of the existing 
SADB list 

To illustrate the use and administration of an FFE, we' 11 go through an 
example using FFE 70 in Figure 3. Firewalls 14 and 18 are both application 
level gateway firewalls implemented according to the present invention. 
35 Workstations H2 and H3 both want to communicate with HI. For the 
administrator of firewalls 14 and 18, this is easy to accomplish. The 
administrator sets up a line something like this (well only show the IP address 
part and SPI parts of the SA, since they're the trickiest values to configure. Also, 
assume that we are using tunnel mode): 

40 

# Hypothetical SW1 Config File 
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# 

# Fields are laid out in the following manner: 

# srcaddrornet- local SPI = peeraddr* peerSPI= 
realsrcaddr= localaddr= key* 

5 

# The following entry sets up a tunnel between hosts 
behind SWl 

# and hosts behind SW2. 

src=172.16.0.0 localSPI-666 peer=192.168.100.5 
10 peerSPl-777 \ 

realsrcaddr=192 . 168 . 100 . 5 localaddrs^O .0.0,0 
key=Oxdeadbeeffadebabe 

# Hypothetical SW2 Config File 
15 # 

# Fields are laid out in the following manner: 

# srcaddrornet= localSPI= peeraddr* peerSPI^ 

realsrcaddr= localaddr** key= 

20 # The following entry sets up a tunnel between hosts 
behind SWl and 

# hosts behind SW2. 

src=172-17.0.0 localSPI*777 peer=192.168.20 .1 
peerSP3>666 \ 

25 realsrcaddr=l92 . 168. 20.1 localaddr=0 . 0 . 0 . 0 \ 

key=Oxdeadbeef fadababe 

With this setup, all traffic is encrypted using one key, no matter who is 
talking to whom. For example, traffic from H2 to HI as well as traffic from H3 

30 to HI will be encrypted with one key . Although this setup is small and simple, it 
may not be enough. 

What happens if H2 cannot trust H3? In this case, the administrator can 
set up security associations at the host level. In this case, we have to rely on die 
SPI field of the SA, since the receiving firewall cannot tell from the datagram 

35 header which host behind the sending firewall sent the packet Since the SPI is 
stored in IPSEC datagrams, we can do a lookup to obtain its value, Below are 
the sample configuration files for both firewalls again, but this time, each host 
combination communicates with a different key. Moreover, H2 excludes H3 
from communications with HI, and H3 excludes H2 in the same way. 
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# Hypothetical SWl Conf ig File 
# 

# Fields are laid out in the following manner: 

# srcaddrornet= localSPI= peeraddr- peerSPI= 
S realsrcaddr= localaddr= key= 

# The following entry sets up a secure link between H2 
and HI 

src-172 - 16 . 0 . 2 localSPI=666 peer=192.16B.100-5 
10 peerSPI-777 \ 

realsrcaddr-192 . 166 .100.5 
localaddrs=l78.17.128.71 \ 
key«*0x0a0a0a0a0a0a0a0a 

15 # The following entry sets up a secure link between H3 
and HI 

src-172.16.0.1 localSPI=555 peer«192.168.100.5 
peerSPI*888 \ 

realsrcaddr=192 . 168 . 100 . 5 
20 localaddrs-178.17.128.7l \ 
key=0x0b0b0b0b0b0b0b0b 

# Hypothetical SW2 Config File 
# 

25 # Fields are laid out in the following manner: 

# srcaddrornet« localSPI= peeraddr= peerSPI« 
realsrcaddr= localaddr- key= 

# The following entry sets up a secure link between H2 
30 and HI 

src=l72.17.12B.71 localSPI=777 peer»192. 168. 20.1 
peerSPI=666 \ 

realsrcaddr-192. 168. 20.1 localaddrs=172 . 16 . 0 . 2 \ 

key=0x0a0a0a0a0a0a0a0a 

35 

# The following entry Bets up a secure link between H3 
and HI 

src=172.17.l28.71 localSPI=888 peer=192, 168. 20.1 
peerSPl«555 \ 

40 realsrcaddr=192 .168.20.1 localaddrB-172.16.0.1 \ 

key=0x0bOb0b0b0b0b0b0b 

Figure 4 is a block diagram showing in more detail one embodiment of 
an EPSEC-cnabled application level gateway firewall 1 8. Application level 
45 gateway firewall 1 8 provides access control checking of both encrypted and 
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unencrypted messages in a more secure environment due to its network- 
separated architecture. Network separation divides a system into a set of 
independent regions or burbs, with a domain and a protocol stack assigned to 
each burb. Each protocol stack 40x has its own independent set of data 
S structures, including routing information and protocol information. A given 
socket will be bound to a single protocol stack at creation time and no data can 
pass between protocol stacks 40 without going through proxy space. A proxy 50 
therefore acts as the go-between for transfers between domains. Because of this, 
a malicious attacker who gains control of one of the regions is prevented from 

1 0 being able to compromise processes executing in other regions. Network 
separation and its application to an application level gateway is described in 
"SYSTEM AND METHOD FOR ACHIEVING NETWORK SEPARATION", 
U.S. Application No. 08/599,232, filed February 9, 1 996 by Gooderum et aL 
In the system shown in Figure 4, the in-bound and out-bound datagram 

1 5 processing of a security association continues to follow the conventions defined 
by the network separation model. Thus aU datagrams received on or sent to a 
given burb remain in that burb once decrypted. In one such embodiment SADB 
socket 78 has been defined to have the type 'sadb\ Each proxy SO that requires 
access to SADB socket 78 to execute its query as to whether the received 

20 message was encrypted must have create permission to the sadb type , 

The following is list of specific requirements that a system such as is 
shown in Figure 4 must provide. Many of the requirements were discussed in 
the information provided earlier in this document 

1 . Firewall applications may query the IPSEC subsystem to determine if 
25 traffic with a given address is guaranteed to be encrypted. 

2. Receipt of an unencrypted datagram from an address that has a SA results 
in the datagram being dropped and an audit message being generated* 

3. On receipt of encrypted protocol datagrams the SADB searches will be 
done using the SPI as the primary key. The source address will a 

30 secondary key. The SA returned by the search will be the SA which 

matches the SPI exactly and has the longest match with the address. 
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4 . A search of the S ADB for a SPI that finds an enlry that is marked as SA 
for adynamic IP will not consider the address in fee search process. 

5. A search of the SADB for a SPI that finds an entry that is marked as a SA 
for a tunnel mode connection will to consider the address if it is (0.0.0.0) 
UINADDR, 

6. On receipt of a non-IPSEC datagram the SADB will be searched for an 
entry that matches the sre address. If a SA is found the datagram will be 
dropped and an audit message sent 

7. SADB searches on output will be done using the DST address as key. If 
more than one SA entry in the SADB has that address the first one with 
the maximum address match will be returned 

8. The SADB must be structured so that searches are fast regardless if the 
search is done by SPI or by address. 

9. The SADB must provide support for connections to a site with a fixed 
SPI but changing IP address. SA entries for such connections will be 
referred to as Dynamic Address Sites, or just Dynamic entries. 

10. When a dynamic entry is found by a SPI search, the current datagram's 
SRC address, which is required to ensure that the return datagrams are 
properly encrypted, will be recorded in the SA only after the AH 
checking has passed successfully. (This is because if the address is 
recorded before AH passes then an attacker can cause return packets of 
an outgoing connection to be transmitted in the clear.) 

11. A failure of an AH check on a dynamic entry results in an audit message. 

12. In an embodiment where the firewall requires that all connections use 
both AH and ESP, on receipt the order should be AH first ESP second. 

13. The processing structure on both input and output should try to minimize 
die number of SADB required lookups. 

Returning to Figure 4, in one embodiment firewall 18 includes a crypto 
engine interface 80 used to encrypt an IPSEC payload. Crypto engine interface 
80 may be connected to a software encryption engine 82 or to a hardware 
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encryption engine 84. Engines 82 and 84 perform the actual encryption function 
using, for example, DES-CBC. In addition, software encryption engine 82 may 
include the keyed MD5 algorithm used for AH. 

In one embodiment, crypto engine interface 80 is a utility which provides 

5 a consistent interface between the software and hardware encryption engines. As 
shown in Figure 4, in one such embodiment interface 80 only supports the use of 
the use of hardware cryptographic engine 84 for IPSEC ESP processing. The 
significant design issue that interface 80 must deal with is that use of a hardware 
encryption engine requires that the processing be down in disjoint steps 

10 operating in different interrupt contexts as engine 84 completes the various 
processing steps. 

The requited information is stored in a request structure that is bound to 
the IP datagram being processed. The request is of type crypto_reques t_t . 
This structure is quite large and definitely does not contain a minimum state set. 

IS In addition to the definition of the request data structure, this software 

implementing interface 80 provides two functions which isolate the decision of 
which cryptographic engine to use. The crypt _des_encrypt function is for 
use by die IP output processing to encrypt a datagram- The 
crypt_des_decrypt function is for use by the IP input processing to 

20 decrypt a datagram. If hardware encryption engine 84 is present and other 

hardware usage criteria are met the request is enqueued on a hardware processing 
queue and a return code indicating that the cryptographic processing is in 
progress is returned. If software engine 82 is used, the return code indicates thai 
the cryptographic processing is complete. In the former case, the continuation of 

25 the IP processing is delayed until after hardware encryption is done. Otherwise 
it is completed as immediately in die same processing stream. 

There are two software cryptographic engines 82 provided in the IPSEC 
software. One provides the MD5 algorithm used by the IPSEC AH processing, 
and the other provides the DES algorithm used by the IPSEC ESP processing. 

30 This software can be obtained from the US Government IPSEC implementation. 
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In one embodiment hardware cryptographic engine 84 is provided by a 
Cylink SafeNodc processing board. The interface to this hardware card is 
provided by the Cylink device driver. A significant aspect of the Cylink card 
that plays a major part in the design of the IPSEC Cylink driver is that the card 
5 functions much like a low level subroutine interface and requires software 

support to iniuate each processing step. Thus to encrypt or decrypt an individual 
datagram there are a minimum of two steps, one to set the DES initialization 
vector and one to do the encryption. Since the IP processing can not suspend 
itself and wait while the hardware completes and then be rescheduled by the 
10 hardware interrupt handler, in one embodiment a finite state machine is used to 
tie sequences of hardware processing elements together. In one such 
embo dimcnt the interrupt handler looks at the current state, executes a defined 
after state function, transitions to the state and then executes that stale's start 
function. 

15 One function, cyl_enqueue_reque 3 t:, is used to initiate cither an 

encryptoradecryptaciion. This function is designed to be catted by 
cryptographic engine interface 80. All of the information required to initiate the 
processing as well as the function to be performed after the encryption operation 
is completed is provided in the request structure. This function will enqueue the 

20 request on the hardware request queue and start the hardware processing if 
necessary. 

A system 30 which can be used for firewall-to-workstation encryption is 
shown in Figure 5. In Figure 5, system 30 includes a workstation 12 
communicating through a firewall 14 to an unprotected network 1 6 such as the 

25 Internet. System 30 also includes a workstation 32 communicaiing directly with 
firewall 14 through unprotected network 16. Firewall Mis an application level 
gateway incorporating IPSEC handling as described above. Qx should be noted 
that IPSEC security cannot be used to authenticate the personal identity of the 
sender for a firewall to firewall transfer. When IPSEC is used, however, on a 

30 singleusermachinesuchasapomblepersoiid 
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be protected with a personal identification number (PIN)- In these cases IPSEC 
can be used to help with user identification to the firewall) 

According to the IPSEC RFC's, you can use cither tunnel or transport 
mode with this embodiment based on your security needs. In certain situations, 
5 the communications must be sent in tunnel mode to hide unregistered addresses. 

Although specific embodiments have been illustrated and described 
herein, it will be appreciated by those of ordinary skill in the art that any 
arrangement which is calculated to achieve the same purpose may be substituted 
for the specific embodiment shown. This application is intended to cover any 
10 adaptations or variations of the present invention. Therefore* it is intended that 
this invention be limited only by the claims and the equivalents thereof. 
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What is claimed is: 

1 . A method of regulating the flow of messages through a firewall having a 
network protocol stack, wherein the network protocol stack includes an Internet 

5 Protocol (DP) layer, the method comprising the steps of: 

determining, at the IP layer, if a message is encrypted; 
if the message is not encrypted, passing the unencrypted message up the 
network protocol stack to an application level proxy; and 

if the message is encrypted, decrypting the message and passing the 
10 decrypted message up the network protocol stack to the application level proxy, 
wherein the step of decrypting the message includes the step of executing a 
procedure at the IP layer to decrypt the message. 

2. A method of authenticating the sender of a message within a computer 
1 5 system having a network protocol stack, wherein the network protocol stack 

includes an Internet Protocol (IP) layer, the method comprising the steps of: 
determining, at the IP layer, if the message is encrypted; 
if the message is encrypted, decrypting the message wherein the step of 

decrypting the message includes the step of executing a process at the IP layer to 
20 decrypt the message; 

passing the decrypted message up the network protocol stack to an 

application level proxy; 

determining an authentication protocol appropriate for the message; and 
executing the authentication protocol to authenticate die sender of the 
25 message. 

3. The method according to claim 2 wherein the step of determining an 
authentication protocol appropriate for the message includes the steps of: 

determining a source IP address associated with the message; and 
30 determining the authentication protocol associated with the source IP 

address. 
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4, The method according to claim 2 wherein the message includes security 
parameters index and wherein the step of detenuining an authentication protocol 
appropriate for die message includes the steps of: 

determining the authentication protocol associated with a dynamic IP 
5 address, wherein the step of determining die authentication protocol includes the 
step of looking up a security association based on the security parameters index; 

determining a current address associated with die dynamic source IP 
address; and 

binding the current address to the security parameters index. 

10 

5. A firewall, comprising: 

a first communications interface; 

a second communications interface; 

a network protocol stack connected to the first and the second 
15 communications interfaces, wherein the network protocol stack includes an 
Internet Protocol (IP) layer and a transport layer; 

a decryption procedure, operating at the IP layer, wherein the decryption 
procedure decrypts encrypted messages received at one of said first and second 
communications interfaces and outputs decrypted messages; and 
20 a proxy, connected to the transport layer of said network protocol stack, 

wherein die proxy receives decrypted messages from the decryption procedure 
and executes an authentication protocol based on die content of the decrypted 
message. 

25 6. A firewall, comprising: 

a first communications interface; 
a second communications interface; 

a first network protocol stack connected to the first communications 
interface, wherein the first network protocol stack includes an Internet Protocol 
30 (IP) layer and a transport layer; 
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a second network protocol stack connected to the second 
communications interface, wherein die second network protocol stack includes 
an Internet Protocol (IP) layer end a transport layer; 

a decryption procedure, operating at the IP layer of the first network 
5 protocol stack, the decryption procedure receiving encrypted messages received 
by said first communications interface and outputting decrypted messages; and 

a proxy, connected to the transport layers of said first and second network 
protocol stacks, the proxy receiving decrypted messages from the decryption 
procedure and executing an authentication protocol based on the content of the 
1 0 decrypted message. 

7, The firewall according to claim 6 wherein the firewall further includes: 
a third communications interface; and 

a third network protocol stack connected to the third communications 
1 5 interface and to the proxy, wherein the third network protocol stack includes an 
Internet Protocol (IP) layer and a transport layer and wherein the second and 
third network protocol stacks are restricted to first and second burbs, 
respectively. 

20 8. A method of establishing a virtual private network between a first and a 
second network, wherein each network includes an application level gateway 
firewall which uses a proxy operating at the application layer to process traffic 
through the firewall, wherein each firewall includes a network protocol stack and 
wherein each network protocol stack includes an Internet Protocol (IP) layer, the 

25 method comprising the steps of: 

transferring a connection request from the first network to the second 
network; 

determining, at the IP layer of the network protocol stack of the second 
network's firewall, if the connection request is encrypted; 
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if the connection request is encrypted, decrypting the request, wherein the 
step of decrypting the request includes the step of executing a procedure at the IP 
layer of the second network's firewall to decrypt the message; 

passing the connection request up the network protocol stack to an 
5 application level proxy; 

determining an authentication protocol appropriate for the connection 
request; 

executing the authentication protocol to authenticate the connection 
request; and 

10 if the connection request is authentic, establishing an active connection 

between the first and second networks. 

9- The method according to claim 8 wherein the step of executing the 
authentication protocol includes the step of executing program code within the 
IS firewall of the second network to mimic a challenge/response protocol executing 
on a server internal to die second network. 

10. The method according to claim 8 wherein the step of execute 
authentication protocol includes the step of executing program code to execute 

20 the authentication protocol in line to die session. 

11. The method according to claim 8 wherein the step of determining an 
authentication protocol includes the step of determining if the connection request 
arrived encrypted and selecting the authentication protocol based on whether the 

25 connection request was encrypted or not encrypted. 
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